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Abstract 

In  this  work,  we  present  a  technique  for  capturing  a  time-varying  human  performance  in  such  a 
way  that  it  can  be  re-illuminated  in  postproduction.  The  key  idea  is  to  illuminate  the  subject  with 
a  variety  of  rapidly  changing  time-multiplexed  basis  lighting  conditions,  and  to  record  these 
lighting  conditions  with  a  fast  enough  video  camera  so  that  several  or  many  different  basis 
lighting  conditions  are  recorded  during  the  span  of  the  final  video's  desired  frame  rate.  In  this 
poster  we  present  two  versions  of  such  a  system  and  propose  plans  for  creating  a  complete, 
production-ready  device. 


1.  Relighting  Using  Interlaced  Video  Patterns 


(a)  (b)  (c) 


Fig  1 .  An  actress  lit  (a)  from  the  left  and  (b)  from  the  right 
by  interlaced  television  patterns,  (c)  A  linear  combination  of  the 
two  images  produces  a  novel  lighting  condition.  A  30fps  sequence 
of  this  result  is  included  in  the  video. 

Our  first  experiment  uses  two  interlaced  television  monitors  showing  patterns  of  just  even  and 
just  odd  lines  to  light  a  performance  alternately  from  the  left  and  right  at  60fps.  A  standard 
interlaced  video  camera  records  the  performance  so  that  the  even  lines  of  the  video  show  the 
subject  illuminated  from  the  left  and  the  odd  lines  show  the  subject  illuminated  from  the  right. 
We  de-interlace  the  video  into  two  30fps  progressive  streams  and  take  linear  combinations  of  the 
lighting  configurations  to  produce  a  novel  illumination  of  the  performance  at  30fps  (Fig.  1). 


2.  Relighting  Using  an  Interleaved  Spherical  Harmonic  Basis 

Our  second  experiment  uses  a  two  meter  sphere  of  inward-pointing  LED  lights  similar  to 
[Debevec  et  al,  SIGGRAPH  2002]  to  rapidly  cast  a  sequence  of  ten  basis  lighting  environments 
on  an  actor's  live  performance.  In  this  experiment,  we  used  the  first  nine  terms  of  the  Spherical 
Harmonic  (SH)  basis,  plus  one  directional  light  source.  Since  all  but  the  first  element  of  the  SH 
basis  are  negative  in  some  directions,  we  offset  and  scale  each  basis  function  to  range  between  0 
and  1,  seen  in  the  bottom  row  of  Fig.  1,  and  the  resulting  images  of  the  actor  are  later  re-scaled  by 
the  inverse  transformations  using  the  DC  component  lighting  image  to  provide  the  offset  amount. 
The  LED  patterns  are  synchronized  with  a  progressive  video  camera  so  that  each  frame  captures  a 
single  element  of  the  lighting  basis.  This  process  yields  a  series  of  ten-frame  sets  we  call 
extended  frames,  such  as  those  seen  in  the  top  three  rows  of  (Fig.  2). 

Using  this  lighting  basis,  we  can  re-illuminate  the  actor's  performance  according  to  a  captured 
lighting  environment  (such  as  the  one  seen  in  the  lower  right  of  Fig.  2)  by  first  computing  the  SH 
basis  coefficients  of  that  lighting  environment.  We  form  linear  combinations  of  the  color 
channels  of  the  basis  images  according  to  these  coefficients  to  produce  the  re-illuminated  images 
similar  to  those  seen  at  the  right  of  (Fig.  2).  The  process  is  similar  to  that  described  in  [Debevec 
et  al,  SIGGRAPH  2000],  except  using  a  different  lighting  basis. 


Fig.  2:  An  actor  stands  within  a  sphere  of  inward-pointing  LED 
lights  as  elements  of  the  spherical  harmonic  lighting  basis 
(bottom)  are  projected  in  rapid  succession  during  each  extended 
frame.  Linear  combinations  of  the  frames  produce  novel 
illumination  on  the  performance  according  to  a  sampled  lighting 
environment  (right). 

Since  the  frames  making  up  an  extended  frame  are  not  taken  at  precisely  the  same  time,  the 
subject  may  move  noticeably  from  frame  to  frame,  causing  ghosting  in  the  re-illuminated 
sequence.  We  compensate  for  the  subject  movement  using  an  optic  flow-based  image  tracking 
program  to  estimate  the  motion  between  the  captured  frames,  and  then  warp  each  image  into 
alignment  with  the  first  image  of  each  extended  frame.  This  largely  eliminates  the  ghosting 
effect.  Since  the  video  camera  used  in  this  experiment  ran  at  60fps,  the  extended  frames  could  be 
captured  at  no  more  than  6fps.  Thus  we  used  the  optic  flow  technique  to  generate  intermediate 
frames  to  create  a  30fps  sequence. 

Our  next  experiment  will  use  a  video  camera  running  at  over  l,000fps  to  record  the  actor  with  a 
larger  set  of  basis  lighting  conditions  per  extended  frame,  and  to  produce  a  30fps  sequence 
without  requiring  frame  interpolation.  The  larger  number  of  basis  lighting  conditions  (which  will 
include  additional  directional  lighting  conditions)  will  allow  environments  with  high-frequency 


lighting  components  to  be  reproduced  more  accurately.  We  are  also  developing  a  user  interface 
to  allow  a  lighting  designer  to  intuitively  adjust  the  final  illumination  manually.  The  result  will 
be  a  sufficiently  high  degree  of  control  for  matching  the  subject  realistically  and  aesthetically  into 
most  illumination  environments. 

Other  interleaved  lighting  conditions  could  provide  additional  data  also  useful  for  relighting  or 
otherwise  transforming  the  actor’s  performance.  These  conditions  could  include  structured  light 
patterns  produced,  for  example,  by  a  video  projector  or  a  laser,  that  would  allow  a  three- 
dimensional  model  of  the  actor  to  be  derived  for  each  frame.  The  conditions  could  also  include  a 
basis  image  in  which  the  actor  is  silhouetted  by  a  field  of  light  behind  them;  such  an  image  could 
be  used  to  form  an  alpha  channel,  or  matte,  for  the  actor  in  order  to  facilitate  compositing  them 
into  a  scene. 
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